Repository navigation
Make the compaction group graceful period configurable per block range - #7916
Merged
yeya24 merged 5 commits intoNov 6, 2026
Merged
Conversation
The partitioning grouper holds a compaction group back until its time range ended more than twice the cleanup interval ago. That graceful period is the same for every block range, so the group for the last sub-range of a larger range expires at the same moment as the larger range's own group. A larger range can therefore start compacting while its last sub-range is still collecting blocks, which splits the result into a group compacted early and a second group for the blocks that arrived afterwards. Add -compactor.block-ranges-compaction-delay, a list matched to -compactor.block-ranges by position, so a larger range can be made to wait for its sub-ranges to settle. Leaving it empty keeps the existing behaviour: every range falls back to twice the cleanup interval, which is also what any range without a configured entry uses. The delay is a lower bound. Groups only become eligible on a compaction run, so the effective wait can be up to the compaction interval longer. Signed-off-by: Ben Ye <benye@amazon.com>
Signed-off-by: Ben Ye <benye@amazon.com>
Signed-off-by: Ben Ye <benye@amazon.com>
Signed-off-by: Ben Ye <benye@amazon.com>
Signed-off-by: Ben Ye <benye@amazon.com>
danielblando
approved these changes
Nov 6, 2026
alexqyle
approved these changes
Nov 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does:
Makes the compaction group graceful period added in #7182 configurable per block range, via a new
-compactor.block-ranges-compaction-delay.#7182 holds a compaction group back until its time range ended more than twice
-compactor.cleanup-intervalago. That period is the same for every entry of-compactor.block-ranges, and the gate only looks atrangeEnd, so the group for the last sub-range of a larger range expires at the same moment as the larger range's own group. With the default2h,12h,24h, the 2h group for[10:00, 12:00)and the 12h group for[00:00, 12:00)become eligible together.A larger range can therefore start compacting while its last sub-range is still collecting level-1 blocks. The blocks that arrived in time are compacted into the larger range's result and marked for deletion; the ones that arrived afterwards can no longer join it and end up in a second, independent group covering the same range. Both results are valid blocks for the same time range, which is fine for querying but does mean a consumer that aggregates blocks per time range sees the same range represented twice.
The new flag is a list matched to
-compactor.block-rangesby position, so a larger range can be configured to wait for its sub-ranges to settle first.Behaviour and compatibility:
2 * -compactor.cleanup-interval. That fallback is also what any range without a configured entry uses, so the coupling to the cleanup interval (and therefore to the bucket index refresh rate, which is the second reason given in Add graceful period for compaction groups #7182) is preserved when the flag is not set.0means no delay, and is distinct from leaving the list empty.Validaterejects a list that is non-empty and does not have one entry per block range, and rejects negative entries.-compactor.compaction-intervallonger. The flag description says so.ShuffleShardingGrouperhas no such gate.Which issue(s) this PR fixes:
NA
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]Per GENAI_POLICY.md: this change was drafted with an AI coding assistant. I have reviewed every line, verified the behaviour against the tests below, and I take responsibility for it under the DCO.
Tests added:
TestPartitionCompactionGrouper_CompactionDelayForRange(per-range lookup, empty-list fallback, configured zero, unknown range, partially configured list),TestPartitionCompactionGrouper_GroupBlocksByRange_CompactionDelayand..._CompactionDelayOfNonFirstRange(the gate end to end for the 2h and 12h ranges, including a group 1ms past its delay),TestPartitionCompactionGrouper_GroupBlocksByRange_DefaultDelayFollowsCleanupInterval(unconfigured still tracks the cleanup interval), andTestConfig_Validatecases for the length and negative-value checks.